Image Difference Captioning with Pre-training and Contrastive Learning

The Image Difference Captioning (IDC) task aims to describe the visual differences between two similar images with natural language. major challenges of this lie in aspects: 1) fine-grained that require learning stronger vision and language association 2) high-cost manual annotations leads limited supervised data. To address these challenges, we propose a new modeling framework following pre-training-finetuning paradigm. Specifically, design three self-supervised tasks contrastive strategies align text descriptions at level. Moreover, data expansion strategy utilize extra cross-task supervision information, such as for image classification, alleviate limitation available IDC Extensive experiments on benchmark datasets, CLEVR-Change Birds-to-Words, demonstrate effectiveness proposed framework. codes models will be released https://github.com/yaolinli/IDC.